Add PMTiles and improve assets links#21
Conversation
jenningsanderson
had a problem deploying
to
staging
GitHub Actions
Failure
jenningsanderson
force-pushed
the
stac-with-pmtiles
branch
from
2b7a8c9 to
7afd66b
Compare
jenningsanderson
temporarily deployed
to
staging
GitHub Actions
Inactive
jenningsanderson
temporarily deployed
to
staging
GitHub Actions
Inactive
jenningsanderson
force-pushed
the
stac-with-pmtiles
branch
from
fc7031c to
ba237fa
Compare
jenningsanderson
temporarily deployed
to
staging
GitHub Actions
Inactive
jenningsanderson
force-pushed
the
stac-with-pmtiles
branch
from
ba237fa to
aa4fe4a
Compare
jjcfrancisco
left a comment
jjcfrancisco
left a comment
There was a problem hiding this comment.
Looks good! I have left optional minor comments. I also have couple of general questions:
- Why is the STAC catalog published daily (i.e.
publish-stac.yaml) if releases are monthly? - Why is the same STAC catalog generated twice one for
labs.overturemaps.org/stac/and one forstac.overturemaps.org/?
| logger = logging.getLogger(__name__) | ||
|
|
||
| # Schema version mapping (same as CLI) | ||
| SCHEMA_VERSION_MAPPING: dict[str, str] = { |
There was a problem hiding this comment.
Shouldn't this be imported directly cli.py instead of duplicating?
from overture_stac.cli import schema_version_mapping
...
schema = schema_version_mapping.get(release, "unknown")There was a problem hiding this comment.
Yes, nice catch! This was thrown together and I want to revisit it's ability to run as part of CI as well.
| fs.FileSelector("overturemaps-us-west-2/release") | ||
| ) | ||
|
|
||
| # TODO: These should be stored elsewhere, but for now we'll hardcode them here |
There was a problem hiding this comment.
If schema versions are available in new STAC catalogs, do we need to maintain this mapping? We could read from the published catalog and only require schema for new releases.
There was a problem hiding this comment.
I thought about this — but then we'd always need to be passing in the new schema version.
I don't think that any information surfaced from STAC should be anything new — it should just be a convenient index into the Overture Release and artifacts. I think we need to tie the schema version to the data somewhere else in the pipeline so that it's discoverable.
|
|
||
| # TODO: These should be stored elsewhere, but for now we'll hardcode them here | ||
| schema_version_mapping = { | ||
| schema_version_mapping: dict[str, str] = { |
There was a problem hiding this comment.
You may also need to change this when instantiating OvertureRelease
| schema_version_mapping: dict[str, str] = { | |
| SCHEMA_VERSION_MAPPING: dict[str, str] = { |
| # Get all fragments | ||
| all_fragments = list(type_dataset.get_fragments()) | ||
| if debug: | ||
| all_fragments = all_fragments[:2] |
There was a problem hiding this comment.
Just curiosity: why debug only gets the two first in the list?
There was a problem hiding this comment.
Just to run faster — there are a lot of files!
It runs daily to pick up any changes in the s3://overturemaps-us-west-2 bucket from the 60 day retention policy, which does not always line up with the release.
We could probably turn this off, but it's nice to have a running version of |
2 participants
Adds PMTiles and uses alternate and storage extensions